aSGD: Stochastic Gradient Descent with Adaptive Batch Size for Every Parameter

In recent years, deep neural networks (DNN) have been widely used in many fields. Lots of effort has put into training due to their numerous parameters a network. Some complex optimizers with hyperparameters utilized accelerate the process network and improve its generalization ability. It often is trial-and-error tune these optimizer. this paper, we analyze different roles samples on parameter update, visually, find that sample contributes differently update. Furthermore, present variant batch stochastic gradient decedent for using ReLU as activation function hidden layers, which called adaptive descent (aSGD). Different from existing methods, it calculates size each model uses mean effective actual updates. Experimental results over MNIST show aSGD can speed up optimization DNN achieve higher accuracy without extra hyperparameters. synthetic datasets redundant nodes effectively, helpful compression.